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© A system for automatic adjustment and editing of handwritten text images. 

© 1- A data input and display system comprises an x-y 
tablet (80) and stylus (84) inputting a handwritten image in 
X-Y coordinate signals for display on a screen or other 
output device. The system includes logic (50, 52, 53 r 54, 56 
and 58) for collocating the coordinate signals into gross 
discrete groups, potentially words but without identifying 
the data significance of such groups, corresponding to the 
structural grouping of the handwritten image input and 
determines from the coordinate signals within such groups 
individually, structural data relating to each such group 
individually and to sets of such groups. From this data, the 
output image is adjusted to linear and spatial uniformity. 
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A SYSTEM FOR AUTOMATIC ADJUSTMENT AND 
EDITING OF HANDWRITTEN TEXT IMAGES 

This invention relates to a system for providing full function 
electronic communication relative to hand prepared images such as 
handwriting and sketches together with key-entered text. In contem- 
porary communications systems, a need exists to accommodate image 
communication as well as keyed in text. Such image communication, 
distinct from image identification, includes the capability of 
transmitting and receiving hand-prepared images. In the context of 
transmission, such a system must have the capability of providing for 
original creation of documents , annotation of existing documents, the 
completion by hand drawn figures. This degree of functionality 
involves data formatting and editing. 

Such systems must be "user friendly" in creating and editing images 
as well as in message transmission. Ideally, the creation of images* 
should be done utilising equipment providing users with the ability 
to capture and display hand drawn images, automatically adjust size 
and position of various image elements, and, alter the image 
utilising easy to perform operations. Moreover, such equipment 
should be available to multiple users at convenient locations, such 
as at desk sites with hardware which is physically unobtrusive and 
can be operated utilising simple procedures which do not interfere 
with the user's concentration on the substantive content of the task 
at hand. 

In order to fulfil these requirements, without disturbing the users' 
concentration, a system methodology must be defined which allows for 
information entry and edit operations which are accomplished with at 
least the same ease as in physically creating the data, i.e., hand- 
writing. This functionality includes text and image recording. 
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selective editing such as erasing, insertion and the ability to 
change the order of the text by strokes of a pen-like device on paper 
or display. 

Reference is made to US Patents 3,761,877 and 3,399,401. These 
patents broadly disclose the concept of employing a computer based 
graphic display system employing an X-Y input tablet together with a 
writing stylus and display on a CRT. In both patents, the computer 
employs circuitry for converting the data for purposes of display on 
the screen but does not provide any techniques for formatting such as 
segmentation, baseline drift correction, and editing. The '877 
patent discloses broadly the concept of character recognition; 
however, recognition per se is not a part of the present invention. 

Consequently, while generalised concepts of employing a X-Y tablet 
for inputting data onto a CRT screen are known, there exists a 
requirement in the technology for a system which is capable of 
separating handwritten data into individual segments or words and 
treating such segments as a block of data material which can be 
formatted and edited. The formatted and edited data may be stored, 
transmitted or later displayed. 

Given the deficiencies in present image communication processors, the 
present invention provides a data input and display system 
comprising: 

x-y tablet and stylus means having an input means for inputting 
a handwritten image and converting into coordinate signals? and 

display means to portray the handwritten image on a display 
screen or other output device; 

characterised in that the system includes 
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means for collocating the coordinate signals into gross discrete 
groups, without identifying the data significance of such 
groups, corresponding to the structural grouping of the 
handwritten image; 

means for determining from the coordinate signals within such 
groups individually, structural data relating to each such group 
individually and to sets of such groups; and 

means for adjusting such groups to cause their structural data, 
individually and by set, to approach conformity with 
predetermined default criteria whereby, in the eventually 
displayed image, the structural groupings thereof are uniformly 
separated and aligned. 

An embodiment of the invention described in detail hereinafter, 
employs an electronic tablet and an all-points-addressable display to* 
create handwritten images of documents which may be easily edited and 
then stored within an information processing system. The data may be 
subsequently transmitted to one or more recipients equipped with 
suitable all-points addressable displays. By utilising editing and 
automatic formatting techniques, the representation of the image data 
in the system and in the output device is altered and presented in a 
more legible manner consistent with 

conventional text presentation. Examples of editing carried out by 
the present invention are deletion and insertion operations on data 
components. Examples of formatting are paragraph indentation, word 
separation, and baseline drift correction. 

The present invention will be described further by way of example 
with reference to a preferred embodiment thereof, as illustrated in 
the accompanying drawings, in which:- 
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Figure 1 is a functional block diagram showing the procedure for 
segmenting uncoded information into component word images; 

Figure 2 is a schematic diagram showing the technique of storing 
extremity information as a part of the word segmentation techniques 
of Figure 1; 

Figure 3 is an illustration showing segmentation within a zone of 
proximity employing a parallelogram as the zone definition; 

Figure 4 is a schematic block diagram of the logic for positioning 
uncoded words in the display image; 

Figure 5 is a block diagram of the technique for baseline drift 
correction; and 

Figure 6, in segments 6a and 6b , is a block diagram illustrating NCI * 
text editing operations. 

In the embodiment of the invention to be described in detail 
hereinafter, the input device is an electronic tablet utilising apa 
(all points addressable) techniques to record handwritten images of 
documents which may be subsequently edited. Such tablets are known 
in this field, generally having either opaque or transparent screens. 
Such devices are high resolution input systems where every point is 
defined by a pair of co-ordinates. Co-ordinates are collected, 
generally at the rate of 100-200/second. The sampling rate employed 
in such apa systems is a function of anticipated movement of the 
stylus . 

Such known electronic tablets have, in the past, been used for a 
variety of purposes. Typical are the preparation of engineering 
drawings of mechanical or electrical devices and for the recognition 
of handwritten characters to produce coded inputs for information 
processing. It is again pointed out that this invention does not 
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deal with character recognition per se , such as techniques defined 
in, for example , US Patent 3 , 111, 646 , IBM Technical Disclosure 
Bulletin Vol 24, No llA r pages 5404 to 5409, April 1982 and IBM 
Technical Disclosure Bulletin, Vol 24, No 6, pages 2897 to 2902, 
November 1981 • 

The tablet signals are processed automatically to improve the image 
display of handwritten words and symbols. 

Typical operations are identifying words as units of presentation for 
handwritten text, adjusting the interval between consecutive words 
and lines of text to provide uniform spacing, correcting skew lines 
of handwriting (baseline drift correction) and presenting handwritten 
text in paragraphs reflecting the shape the user has created on 
paper. 

In accordance with known techniques, during data creation, signals 
are created utilising tablet entry such that a series of digital 
values corresponding to the co-ordinates of successive positions are 
generated as the pen moves across the tablet surface under user 
control. Digital data is produced as pen movement occurs across the 
surface. Such is accomplished by means of frequent sampling and 
subsequent filtering in a manner designed to retain significant shape 
and location data. 

The fundamental elements of such tablet input data are the X-Y 
co-ordinate pairs. These are combined to form a stroke , that is, 
substantive pen movement. A stroke is defined as a sequence of 
points between the pen initially striking the tablet in a downward 
motion and the pen being lifted up. These co-ordinate pairs form the 
basic units of tablet output which are subsequently processed. Pen 
strokes may be employed to perform various functions. These are 
typically information entry, command entry and operand identifica- 
tion. A menu point can be predefined such that information is 
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entered by touching it when its co-ordinates are produced, the system 
is programmed to record a prescribed message* Alternatively r the 
image information can be entered by drawing it with the pen on the 
tablet. Commands can be entered by touching menu points or by making 
characteristic marks on the tablet. In order to sense these marks 
the system must be programmed to recognise them. If an image item is 
an operand for command operation that is being performed, it can be 
identified by touching the item with the pen where it appears on the 
tablet. When the pen touches one or more points which fall within an 
image item, that item is identified for the operation. It is under- 
stood that known programming techniques are employed to create menu 
displays and processing logic operation in response to menu selec- 
tion. 

Logic is provided to automatically process tablet entered inages for 
detection of fundamental features of those images. Such images 
correspond to symbols, words, lines and paragraphs which make up a * 
document image. The methodology of such logic is to compare each 
succeeding tablet signal with the signals that have preceded it. 
Points where the pen is lifted are detected and displacements between 
pen down and pen up are used to determine the limits of image 
features. Running values are retained by the system for feature 
extremities and a baseline is estimated for text images. The posi- 
tion of each succeeding signal is them compared with preceding 
extremity information to determine where the previous word or line 
has ended and a new symbol has begun. By storing these points of 
segmentation with the recorded signals, boundaries and relative size 
of symbols, words and lines can be identified in the stored image 
data. 

Once segmented, tablet image data is them employed to create images 
which have a more uniform separation between words and additionally 
fill the image display area effectively. In instances where words 
are written such that the characters are significantly larger then 
others in the document, they can be reduced to fill the space more e 
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effectively. As additional words are added to the written record, 
additional lines are appended to the image display. In situations 
where the added words are written directly following prior words on 
the tablet, new display lines will be a part of the same paragraph. 
In situations where added words are indented and the priority line 
did not start a new paragraph, lines are skipped, or an added word 
does not fill a prior line, the output image will start as a new 
paragraph. These operations occur as a function of the formatting 
algorithms hereof. 

The segmented tablet image data is also used to correct baseline 
excursions which occur when the writer does not follow a horizontal 
line across the page. As is well known, handwriting often runs 
either "uphill" or "downhill", in either case a deviation from the 
horizontal. An estimated baseline is determined for each line of 
words on the tablet. This arrangement allows for position adjustment* 
in both the displacement and rotational orientations with respect to 
the baseline. The formatting methodology accomplishes these tasks. 

Editing of handwritten text which is retained in non-coded form is 
another application of segmented tablet image data. Editing opera- 
tions include the ability to delete, replace, insert or move portions 
of non-coded text in a handwritten page. These functions can be 
initiated by utilising the pen to touch a menu point whereby 
identification of the action to be taken occurs. This is then 
followed by touching with the pen the text locations where the action 
is to occur. 

Referring now to Figure 1, as schematic diagram of the technique for 
segmenting non-coded information (NCI) line images is depicted. X 
and Y signals from the writing tablet indicate the current position 
of the pen on the tablet given the all points addressable nature of 
the system. The D input signal indicates when the pen touches the 
tablet. As shown in Figure 1, the beginning of each NCI segment is 
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initiated by a pen down signal, that is r the initiation of a "D H 
signal to gate 10. While the pen is down, and in contact with the 
tablet, X and Y signals are generated and passed to stroke extremity 
detectors 12, (see Figure 2) . Four stroke extremity detectors are 
employed. One is for determining left extremity, a second for right 
extremity, a third for upward extremity and a fourth for downward 
extremity. The X and Y signals from the tablet are passed to all of 
these detectors and appropriately stored. 

Referring to Figure 2, a schematic diagram of one such extremity 
detector is illustrated. The example shown is for a right extremity 
detector, it being understood that the remaining three operate 
functionally in this same manner. As each new point occurs, its X 
and Y co-ordinates are compared with those previously stored which 
have be determined to be extreme values. In this case only X values 
are considered. If the X value exceeds the stored, predetermined 
right extreme value by a predetermined threshold (t) thus indicating 
movement to the right, a new extreme value is detected- The thres- 
hold (t) is employed to accommodate for pen jitter. Thus, as shown 
in Figure 2, the input X signal is first subjected to a comparison 22 
to determine whether it is greater than the current extreme value (e) 
by a predetermined amount, that is, is it greater or equal to the 
threshold (t) . If the indication is positive, the X value is gated 
to the AND gate 24 shown in Figure 2 and stored in register 26 as a 
new extremity value for X. Subsequent X values are compared and 
updated with the newly stored right extreme value. 

In a compatible manner, extreme values of (-X, + Y and -Y) are 
compared with a threshold and extreme values are stored. 

Referring back to Figure 1, whenever an end of stroke is detected as 
a function of pen lift, that is movement of the pen off the tablet, 
the value stored in each of the stroke extremity storage registers 26 
are passed to a proximity zone compare system 14 through gate 16. 
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Stroke extremities are compared with prior word extremities to test 
whether the new stroke lies outside the proximity zone and therefore 
a new word is begun. When a new word is begun a new word signal is 
generated clearing new word gate 18 to pass extremity values for the 
new word. When the old word is continued, stroke extremity values 
are passed to the word extremity detector 17 utilising gate logic 
such as shown in Figure 2. Updated extreme values for the old word 
are then stored in the word extremity store 19 shown in Figure 1. 
When a new word has begun, the word extremity store is erased and 
extremities of the first stroke of the new word are then passed from 
the word extremity detector to the word extremity store 19. 

As illustrated in Figure 1, the proximity zone compare 14 is used to 
determine the beginning of new words for purposes of defining a zone 
of proximity for each word that is being segmented. The zone or 
proximity may have any arbitrary shape, however, a parallelogram is * 
the easiest to implement. An example of a suitable parallelogram 
shown in Figure 3, is a rectangle. Each new stroke is compared with 
the zone of proximity of the current word. If any part of the stroke 
falls inside the zone, it is considered to be part of the current 
word. If it does not fall within the zone, it is considered to be 
either the beginning of a new word or the continuation of a word 
previously thought to be completed. 

Figure 3 illustrates segmentation utilising a rectangular proximity 
zone. It should be understood that any parallelogram may be em- 
ployed. In some alphabets or in cases where letters are slanted a 
different zone shape may be used. The rectangular zone illustrated 
in Figure 3 is defined by horizontal lines passing through the upper 
and lower extremities and vertical lines that pass through the left 
and right e extremities. A new word is detected when a stroke occurs 
outside the proximity zones of preceding words. The new word strokes 
must be positioned more than a prescribed threshold distance (denoted 
herein as M r M ) beyond the previously stored extreme right position or 
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beyond the stored extreme left threshold distance (denoted herein as 
H 1 H ). In a corresponding manner, new word strokes must be positioned 
relative to previously stored up and down threshold distances 
(denoted herein as "u" for extreme upward position, and "d" for 
extreme downward position) . The values of these prescribed thres- 
holds "r" , W 1 M , "u" and "d" are based on selected characteristics of 
the NCI images. 

As shown in Figures 3, "h" is denoted as the height of handwriting. 
It may be assumed for English handwriting that "r tt is approximately 
0.5 of "h" while "1" is slightly less than 0.5 of "h" because 
handwriting progresses from left to right under normal conditions. 
Similarly, M d H can be established by inspection to be approximately 
0.667 of "h" while "u" is slightly less than 0.667 of w h w . These 
threshold values must be adjusted when different languages are 
employed. That is, the values described herein are usable for 
English language handwriting. However, if other languages are 
employed, the parameters would have to be suitably adjusted. 

In addition to this technique of word segmentation, word segmentation 
can also be initiated by the occurrence of prescribed external 
events. As indicated, segmentation is a function of spatial 
relationships which in turn are a function of the language employed. 
Segmentation is not necessarily time limited, however, it can be 
determined based on the passage of time in which no additional 
strokes are made. Thus by time monitoring the occurrence of D 
signals, segmentation of data into assumed words may occur. 
Segmentation could also occur by having the writer touch a menu point 
entry when a word is complete. This latter technique, while assuring 
positive segmentation, slows down data entry. 

Word segmentation as illustrated relative to Figures 1 to 3 could 
occur on one line of data at a time. If enough processing memory 
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were available, conceivably segmentation could be delayed until the 
data representing the entire tablet had been loaded into storage, 

After the NCI images have been segmented into defined areas repre- 
senting words, they may be assembled with prescribed formatting rules 
to form documents that can be presented as useful display images, 
transmitted, or printed-out. Each document is developed as a 
collection of words that form one or more paragraphs. Formatting 
rules thus serve to determine the position of each segmented word in 
the shape of a paragraph in the document. The occurrence of a new 
word may require that it be placed at the beginning of a new 
paragraph, appended to the end of an old paragraph or, in some 
situations, inserted within an existing paragraph. When a word is 
appended at the end of a paragraph, or inserted into the middle, its 
closest upstream word in the document image is found and the new word 
is placed immediately following it. When the word is the first word 
of a new paragraph, in accordance with the formatting instructions of* 
this arrangement, it is placed at the beginning of a new line, with 
an identification that indentation is required. 

In order to apply these formatting rules, logic must detect the 
occurrence of new paragraphs. If a word is the first word of 
document then it is by definition the beginning of a new paragraph. 
If a word is the first word of a line that is not the first line of 
the document, it may be the start of a new paragraph or simply the 
continuation of the preceding paragraph. A new paragraph is 
generally detected when the word is indented, outdented or is placed 
well below the prior line, which has not already been determined to 
the first line of a paragraph, or alternatively, at the beginning of 
a new line when the previously line space was not filled. 

Any of these three occurrences can be used to trigger a paragraph 
start. 
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With the words and paragraphs that make up a document detected, 
document content may then be mapped into an image for display or 
print-out. To produce the desired output , this arrangement provides 
an algorithm to fill the available space on each line with properly 
spaced words until all words of the paragraph have been used. It may 
be necessary to fill out lines with words from a subsequent line on 
the tablet and at other times it may be necessary to spill extra 
words from the end of one tablet line into the following lines for 
purposes of display. Again, it is noted that this arrangement makes 
no attempt to recognise the substantive content of the words being 
formatted. Referring now to Figure 4, the logic for positioning NCI 
words in the display image is illustrated in a schematic diagram. 
The inputs to this logic comprise: 

the new word signal of Figure 1? 

x and y co-ordinates of the new word obtained from Pn Down Gate* 
10 of Figure 1? and 

the new word extremity values from Figure 1. 

After the new word signal occurs, from processing of the preceding 
word, the new word's x and y co-ordinates are stored in the New Word 
XY Store 37. As each succeeding new word signal occurs, the contents 
of New Word XY Store 37 are passed into Tablet Word Position Store 
45, where the signals of each word are recorded, together with a 
document word sequence number which is used for subsequent image 
editing operations to be described later. 

The New Word Translate Unit 38 also receives new word extremities of 
x and y from the Word Extremity Store 19 of Figure 1, The New Word 
Translate unit 38, which is connected to New Word XY Store 37 r 
subtracts the minimum value of x from the x co-ordinates of the new 
word and the minimum value of y from the y coordinates of the new 
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word in order to produce a new word representations in X* and Y 1 
relative to an origin at the bottom left corner of the new word. 

New word extremities are also passed to the Word Extremity Detector 
39, which compares the new word extremities with previously stored 
left, right and bottom extremities of the preceding words of the 
document. If a new word extremity exceeds the previously stored 
corresponding extremity, the new value is stored in Extremity Store 
40a, 40b or 40c when the subsequent new word signal occurs. Until 
then, prior values are stored in 40a, 40b, and 40c. 

New word extremity values are also passed into the New Extremity 
Store 35. The extremity values are held in New Word Extremity Store 
35 until the next new word signal occurs. Then they are passed into 
the Prior Word Extremity Store 36, and extremity values for the 
subsequent word are stored in New Word Extremity Store 35. 
When a new word signal occurs, the new word extremities are compared 
with the contents of: 

Prior Word Extremity Store 36; 

Left Extremity Store 40a; Right Extremity Store 40b; and 

Bottom Extremity Store 40c, 

by the New Word Movement Compare Logic 41. Comparisons of these 
values determine if the new word: 

follows the preceding word on the same line; 

starts a new line with no indent; 

starts a new line with indentation; 

starts a new line to the left of prior lines; or 
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skips a line. 

The conditions for the se occurrences are as follows: 

a) Skip Line 

Upper extremity of new word more than "t* beneath lower 
extremity value in Bottom Extremity Store 40c, where "t" is 
approximately twice line height. 

b) New line, no indent. 

Difference between Left Extremity Store 40a contents and 
left extremity of new word less than H m", and upper 
extremity of new word is more than "t/4* beneath lower 
extremity value in Bottom Extremity Store 40c, where "m" is 
a predetermined constant for determining indentation. 

* 

c) New line, indent 

Difference between Left Extremity Store 40a contents and 
left extremity of new word less than M -m w , and upper 
extremity of the new word is more than B t/4" beneath lower 
extremity value in Bottom Extremity Store 40c. 

d) New line, left of margin 

Difference between Left Extremity Store 40a contents and 
left extremity of the new word more than M m" , and upper 
extremity of the new word is more than n t/4 M beneath lower 
extremity value in Bottom Extremity Store 40c. 

e) Same line, following prior word. 

Absolute value of difference between upper extremity of new 
word and upper extremity value in Prior Word Extremity 
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Store 36 less than "t/4", and left extremity of the new 
word is greater than right extremity contents of Prior Word 
Extremity Store 36 by an amount between w m/8 n and M m". 

f) New line, no indent, short prior line. 

Right extremity value of Prior Word Extremity Store 36 more 
than n 2m" less than right extremity value in Right 
Extremity Store 40b, the difference between the contents of 
Left Extremity Store 40a and left extremity of the new word 
less than"m" , and upper extremity of the new word is more 
than" t/4" beneath lower extremity value in Bottom Extremity 
Store 40c. 

If conditions: 

a) skip line; c) new line, indent? d) new line, left of margin;* 
or f) new line, no indent, short prior line; 

occur and the preceding line was not the beginning of a new 
paragraph, a new paragraph condition is indicated by the New 
Paragraph Detector 42. 

• If conditions: 

b) new line, no indent; or c) same line, following prior word; 

occur, New Paragraph Detector 42 is reset and its output indicates an 
old paragraph condition. New paragraph and old paragraph conditions 
are passed to the Display Image Position Control Logic 43, where, 
with new word extremity information, it determines the proper 
location for each word in the display presentation. As each position 
is determined it is stored in the Prior Word Position Store 44, where 
its coordinates are used by Display Image Position Logic 43 to 
determined the display position of the following word. 
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Other inputs to the Display Image Prosition Control Logic 43, 
include: 

the line increments for desired separation in the display 
between left and right margins and for vertical distance between 
lines; and 

the display indent increment (if indent is used) . 

These values are set by the choice of display image size and remain 
constant for a given display geometry. When a new display page image 
is begun, the first word will be positioned at the left edge of the 
top line unless the word is designated as the first word of an 
indented paragraph. If the Display Image Control Logic 43 determines 
the word should be indented it adds the appropriate displacement to 
its horizontal position. When subsequent words occur, the Display 
Image Position Control Logic 43, checks the old or new paragraph m 
status to determine where the words should be positioned. For words 
in old paragraphs, the Display Image Control Logic 43, determines if 
the space between the right edge of the prior word and the right 
margin is large enough to display the new word. If the space is not 
large enough the word is placed at the left edge of the next line by 
decrementing the vertical displacement down one line separation, and 
setting the horizontal displacement at the left margin. If the new 
word starts a paragraph, the vertical displacement is decremented two 
lines separations, and the horizontal displacement is at the margin 
or indented from the margin depending on the display format that is 
chosen by the users. 

The minimum x and y coordinates for the display word position are 
passed to the Display Image Assembly Unit 46, where they are attached 
to the representation of the word in coordinates X' and Y* from New 
word Translate unit 38. These coordinates are passed to Display Word 
Position Store 47 , where they are stored with the same document word 
sequence number that was assigned for reference in the Tablet Word 
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Position Store 45. The Display Word Position Store 45, also stores 
new paragraph indications, line increment information, and line 
indent information with each word that begins a new paragraph. This 
information is obtained form the Display Image Position Control Logic 
43 with the words that begin paragraphs- The Display Word Position 
Store is the source of data that drives the CRT and places the basic 
word shapes at the desired location in the image display. Referring 
now to Figure 5, the technique of baseline correction will be 
explained. A common characteristic of handwritten text is the 
deviation of words an characters from a nominal baseline. When this 
occurs, both the appearance and the interpretation of the text may be 
unacceptable. With the NCI images segmented into words, deviations 
may be corrected for the purpose of image display and subsequent 
interpretation. This arrangement employs a procedure that 
establishes a left to right sequence of estimates for the Y 
coordinate of a nominally established baseline. Two levels of 
baseline correction are accomplished. The first is the correction of 
the word orientation such that each word becomes horizontal. The 
second is the correction of the vertical drift such that baselines 
of the words in one line lie on the same horizontal line. The first 
is called the baseline slant correction and the second is called the 
baseline drift correction. Baseline correction is based on the 
observation that most of the symbols used in a particular alphabet 
sit on a baseline with only a few exceptions. In the case of the 
English language, lower case letters M ff, g, j, p, q and y" and 
sometimes M z M have descenders and do not sit on a baseline. Other 
marks, for purposes of punctuation and the like, such as apostrophes 
and quotation marks sit above the baseline. 

Baseline correction is also a function of word length. When a word 
is short, for example "a, it, of," the baseline can be assumed to be 
horizontal, but could be slightly offset from the running baseline of 
the line being written. Thus, for short words, baseline slant 
correction is not necessary. Before a new baseline is estimated, it 
must be determined whether the word is to sit on a baseline. Recall 
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that words which descenders do not sit on a baseline. This is 
accomplished by testing the minimum Y value of the word against the 
running baseline* If the difference is within a tolerance, for 
example, approximately 0.33 of the line height, then the word is 
assumed to sit on a baseline and a new baseline is computed. 
A new baseline for a word may be computed by determining the baseline 
as equalling WEIGHT * (running baseline) + (1 - weight) * Y min. In 
this computation, Y min is the minimum Y value of the word and the 
"weight" is a constant, approximately 0.5. For conservative 
correction, the weight may be greater than this value and for more 
correction the weight will be slightly less. 

With reference to word orientation, when the word is long, whether it 
contains symbols with descenders or not, symbols without descenders 
sit on a baseline. The method is then to find the baseline and 
correct it to conform to the horizontal for purposes of display. The 
methodology is shown in Figure 5. An uncorrected word is supplied to* 
a word length detector 50. A determination is made of the length of 
the word based on a predetermined number of characters. If the word 
length falls above the predetermined minimum number of characters, 
then it is considered a "long word" and a word baseline is estab- 
lished for that word utilising a word baseline determinator 53. 
This is accomplished by drawing boxes of width about the same as a 
single symbol and high enough to contain all of the points within 
this particular width. Tall boxes contain ascenders/descenders of 
the particular alphabet while short boxes contain ligatures. Boxes 
of medium height contain symbols without ascenders/descenders an 
those sit on the baseline. A selection is then made of prede- 
termined number or boxes of medium height and a determination is made 
by curve fitting of the bottom coordinates to provide baseline slant 
data. With the slant amount thus calculated, Y coordinates are 
corrected to make the word horizontal. 

Once long words are made horizontal and baseline for all words are 
detected, words must be moved vertically to match the running 
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baseline of the line under consideration. Thus, the word baseline is 
supplied to a register 54 of Figure 5 for the determination of a new 
baseline. 

When a first word of a line is written, there is of course no running 
baseline. A lined paper or indication by the writer can be used to 
provide the initial running baseline. Otherwise, the baseline of the 
first word (the calculation of which is described below) is used 
initially, and then may be later modified as more words are written, 
if it happened to be a short word. This is accomplished by using the 
running baseline register 56. An approximate initial value is 
assumed on some basis, lined paper, user indication, or the baseline 
of the first word. As new words are written, register 56 is 
repeatedly updated as shown in Figure 5. 

In the case of a short word being the first word of the line, an 
approximation of the initial running baseline is established using 
the bottom of the word. As shown in Figure 5, it will be recomputed* 
after succeeding words are written. If a long word is the first 
word, its baseline becomes the initial running baseline. This 
initial baseline is not recomputed. Thus, as shown in Figure 5, 
based on the estimate of correction of the running baseline, the word 
is then transformed in register 58 to conform to the running baseline 
and the base-line correct word is issued such that it is displayed on 
the running baseline. As shown in Figure 5, baseline correction 
occurs on a word by word basis as the line is established. Baseline 
correction can also be done at the end of a line. Given adequate 
processing speeds, a line may be written on the input tablet with 
display occurring one line at a time, with sufficient data obtained 
in that period that line by line baseline correction can be attained. 
It is also possible to postpone display until several lines are 
generated such that multiple lines can be corrected at once. 

While these functions have been described using English language as 
the input, it will be understood that the invention can be used with 
other languages. The same methodology would be employed but the 



YO9-83-020 



20 



0176715 



values of selected constants would be chosen consistent with word 
length, characters, etc of the particular language. 

As described herein, a portion of this arrangement concerns 
methodology for formatting handwritten characters for display in an 
organised manner without attempting to recognise the substantive 
content of those characters. This arrangement also provides for 
algorithms used to edit the NCI text. Editing involves changes that 
move, delete or replace image segments which have been partitioned by 
the word segmentation operation shown in Figure 1 and positioned in 
the display by the operations shown in Figure 4. The editing 
operations to be performed can be specified either by touching a menu 
point on the tablet or by making specific characteristic marks for 
the action desired. Editing involves several different types of 
operations and the characteristic mark method would therefore require 
a machine recognition of different types of marks, that is to say, 
the recognition of instructions rather than the recognition of data. 
Such can be accomplished using well known programming techniques for 
the input tablet and pen system, such as pattern recognition 
techniques. 

In addition to techniques to specifying the editing commands to be 
performed, the system also provides for identification of the seg- 
ments which are to be operated on, that is the operands. Operands 
must be identified by pointing to them with the pen, moving a cursor 
or some other physical technique because the uncoded image segments 
cannot be identified by their content. Again, known programming 
techniques for editing using the pen or cursor on the writing tablet 
accomplish this task. For example, the identification of the NCI 
word operand can be made by simply pointing to it with a pen or to 
the space preceding or following it such that delineation is made of 
the required operation, that is, delete, insert or move. When the 
image on the tablet contains the current representation of the image 
which is to be edited, the user may simply point to the required 
operand locations with a pen. 
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However, when the image which was recorded has been stored, operands 
can be identified by calling up the image on the display screen and 
pointing with the pen if a touch sensitive display panel is employed. 

The system logic relating to the position touched on the overlay as 
corresponding to a position on the document is known in this techno- 
logy. Direct pointing to the operand provides a natural and fast 
technique for entering operand information. 

Another editing operation for handwritten text is the addition of 
missing strokes to a word. A common example is the crossing of "t". 
As long as the word is still found on the input tablet, word comple- 
tion strokes within the word f s proximity zone may be recognised and 
added to that word for display as a part of the completed word. If, 
however, the stroke goes beyond the proximity zone, it may cause a 
situation where neighbouring words merge. Word segmentation will 
still occur after the completion of all strokes. * 

Another editing technique is to write over a word for purposes of 
changing that word once it has been written. The arrangement allows 
this editing operation to occur by a mode of operation wherein a new 
stroke can be written over all or part of and existing NCI segmented 
word to replace all or part of the word which is overwritten. This 
may be used for operations such as correcting spelling or breaking a 
long sentence into two with capitalisation of the first word of the 
second sentence. 

Another purpose of writing over a recorded word would be to erase it. 
A characteristic M scratch-out M stroke which never occurs in hand- 
writing may be reserved for such word deletion operations. This can 
be made, for example, by recognising a stroke on the top df the word 
where they are written on the tablet so that the word which has been 
covered by this stroke will be identified and deleted. This would 
correspond to the common practice of simply crossing words out when 
they are handwritten. 
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Referring to Figures 6a and 6b, the method for three basic NCI text 
editing operations, delete, insert, and move, are described. In this 
example the command for each is invoked by touching a "function key" 
area 82 on the tablet. The coordinates from Pen Down Gate 10, Figure 
1 are evaluated by Edit Key Coordinate Analyser 61. If the co- 
ordinates match the values for a delete operation, the Delete Gate 62 
is enabled and subsequent pen coordinates are directed to the Delete 
Operand Register 66. The contents of Delete Operand Registered 66, 
are compared with the coordinates stored in Tablet Word Position 
Store 45, by Delete Compare Logic 69, to determine the document word 
sequence number(s) of the word(s) to be deleted. The document word 
sequence number acts as a pointer to the words in the Display Word 
Position Store 47, that are to be deleted. If the deleted word was 
the beginning of a paragraph, the new paragraph indicator is passed 
to the following word. Delete Compare Logic 69 causes the display 
word positions and the paragraph, indent, and line increment informa- 
tion for the word that precedes the deletion and for the series of * 
words that follow it to pass from Display Word Position Store 47 to 
the appropriate inputs in Figure 4 for new word extremities and prior 
word position in the Word Extremity Detectors 39, New Word Extremity 
Store 35, Display Image Position Control Logic 43 and Prior Word 
Display Position Store 44. Y' Y' coordinates for each of the words 
that follow the deletion are passed to the Display Image assembly 
Unit 46. 

Working in the manner indicated in Figure 4, Display Image Position 
Control Logic 43, and the Display Image Assembly Unit 46 record a new 
display image in Display Word Position Store 47, that does not 
contain the deleted words. 

When the insert command is invoked by touching the insert function 
key. Insert Gate 64 is enabled by the output of the Edit Key Co- 
ordinate Analyser 61 and the following pen output, corresponding to a 
touch in the tablet composition area representing the point where the 
insert is to be made, is recorded in the Insert Operand Register 68. 
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The Insert Compare Logic 71 determines the word sequence numbers 
associated with the position of the insert in Tablet Word Position 
Store 45, and thereby identifies the position in the Display Word 
Position Store 47 where new images will be inserted. After the 
insert* position has been established, a new image is written in the 
auxiliary Scratch Pad 83 area of Tablet 80. Edit Key Coordinate 
Analyser 61 determines that the writing is in the scratch pad area 83 
and enables Auxiliary Record Gate 65, to pass the new word ectre- 
mities from Figure 1, and the X 1 Y« coordinates from New Word Tran- 
slate 38 into Auxiliary Word Position Store 73, under control of 
Auxiliary Record Logic 72. The Auxiliary Record Logic 72 translates 
the lower left extremity coordinates of the first new word to the 
values of the insert position in the Display Word Position Store 47, 
and each subsequent word in the auxiliary record is displaced from 
that initial location by the amount of its displacement in the 
scratch pad area. To complete the insert operation, the information 
for the new words in the Auxiliary Word Position Store 73, are passeS 
to appropriate inputs in Figure 4 for new word extremities and prior 
word position, and the Display Image Position Control Logic 43, and 
the Display Image Assembly Unit 46, position the insertion in Display 
Word Position Store 47, in the manner indicated in Figure 4. The 
words that were behind the insertion position in the Display Position 
Store 47, prior to the insertion, must be repositioned to follow the 
end of the inserted words. The Insert Compare Logic 71, translates 
the lower left extremity coordinates of these words to a position 
behind the last word of the insertion, and then the modified values 
for these words in the Display Word Position Store 47, are passed to 
the Display Image Position Control Logic 43 and Display Image 
Assembly Unit 46 for repositioning in Display Word Position Store 47. 

When the move function key is touched, Move Gate 63 is enabled and 
coordinates of subsequent touches are collected in Move Operand 
Register 67. The first pair of touches define the segment of the 
handwritten page that is to be moved. The coordinates of the third 
touch define the location where the moved words are to be inserted. 
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The Move Compare Logic 70, determines the word sequence numbers that 
correspond to these positions by referring to the information stored 
in Tablet Word Position Store 45. When the move is made, Move 
Compare Logic 70, translates the lower left extremity of the first 
word to be moved to the coordinates of the insertion location in 
Display Word Position Store 47. Each following word to be moved is 
displaced enough to maintain relative word positions in the Display 
Word Position Store 47. The words that are not moved by the move 
operation, per se, are subsequently moved automatically to fill the 
space voided to restore correct relative positioning. Move Compare 
Logic 70, then translates the lower left extremity coordinates of the 
first word after the insert location to a position just beyond the 
right extremity of the last word that is moved. All subsequent words 
are displaced a corresponding amount to retain their relative 
positions in the Display Word Position Store 47. 

Thus, it is apparent that this arrangement provides for basic tasks * 
of text editing, that is moving and deleting characters together with 
basic formatting for organisation of those characters used for 
subsequent storage and display. 

It is apparent that modifications of this invention may be practiced 
with out departing from the scope of the appended claims. For 
example, as delineated in Figure 6a, the tablet used for information 
entry is opaque. In some modes of utilisation entry of information 
is more easily effected by using a display overlay. Those modi- 
fications employ a different data entry technique. 

It should be understood that, in such a scheme, the functions of the 
Display Word Position Store 47 and the Tablet Word Position Store 45 
are nearly identical and could be implemented with a single unit. 

The tablet organisation of Figure 6a is illustrative of one format. 
It is apparent that the command functions could be obtained by use of 
an auxiliary device, i.e keys outside the tablet area- 
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CLAIMS 

1. A data input and display system comprising: 

v x-y tablet (80) and stylus means (84) having an input means for 
inputting a handwritten image and converting into coordinate 
signals; and 

display means to portray the handwritten image on a display 
screen or other output device; 

characterised in that the system includes 

means (35, 36, 37, 39 and 40) for collocating the coordinate 
signals into gross discrete groups, without identifying the data 
significance of such groups, corresponding to the structural 
grouping of the handwritten image; 

means (50, 52, 53, 54 and 56) for determining from the 
coordinate signals within such groups individually, structural 
data relating to each such group individually and to sets of 
such groups; and 

means for (58: generally Figure 6) adjusting such groups to 
cause their structural data, individually and by set, to 
approach conformity with predetermined default criteria whereby, 
in the eventually displayed image, the structural groupings 
thereof are uniformly separated and aligned. 

2. A system as claimed in claim 1 wherein the means for formatting 
comprises means for segmenting one word of the hand- 
written image from another word of the handwritten 
image . 



YO9-83-020 



26 



0176715 



3. A system as claimed in claim 2 wherein the means for segmenting 
comprises 

register means (19) to store extremities of strokes during a 
period of time when the stylus is in contact with the tablet, 

comparator means (22) to determine whether a succeeding stroke 
exceeds the stored stroke extremity positions by a predetermined 
amount, and 

means (24 and 26) to determine whether a succeeding stroke 
exceeds a predetermined zone of proximity that exceeds the 
stored extremity positions, to determine whether a succeeding 
stroke is a continuation of a handwritten word or the beginning 
of a succeeding word. 

4. A system as claimed in any preceding claim, wherein the means 
for formatting comprises 

means (63, 67 and 70) for completing a line of the hand-written 
image with succeeding words; and 

means (42) for determining when a word begins a new paragraph 
and for either indenting or outdenting the word on a new line or 
skipping a line and beginning a new sentence with that word. 

5. The system of claim 4 wherein the means for formatting a new 
paragraph comprises 

detector means (39) for determining, based on coordinates 
signals, extremity positions of new words as they are input to 
the X-Y tablet, extremity store means to store extremity values 
representing maximum left, right and bottom positions of input 
words ; 



YO9-83-O20 



27 



0176715 



means (36) to store extremity values of a prior input word; and 
comparison means (22 , 24 and 26) for determining whether an 
input word follows a preceding word on the same line, starts a 
new line without indenting , skips a line or starts a new line 
horizontally displaced with respect to prior lines. 

6. A system as claimed in Claim 5, wherein the comparison means 
includes logic means for comparing the upper extremity value of an 
input word with the stored maximum bottom extremity value, the logic 
means producing an output for skipping a line when the upper 
extremity value exceeds the stored bottom extremity value by more 
than a predetermined amount. 

7. A system as claimed in claim 5 or claim 6, wherein the 
comparison means includes logic means for comparing the difference 
between a store left maximum extremity value and a left extremity * 
value of an input word with a predetermined value, the logic means 
producing an output for beginning a new line without indentation when 
the comparison is less than the predetermined value. 

8. A system as claimed in any of claims 5, 6 or 7, wherein the 
comparison means includes logic means for comparing a upper extremity 
value of an input word with a stored upper extremity value of a prior 
word r the logic means producing an output for placing an input word 
on the same line as the preceding word when the comparison results 
are then a predetermined value and the left extremity of the input 
word is greater than the stored right extremity of the prior word by 
a predetermined amount. 

9. A system as claimed in any of claims 5,6, 7 or 8, wherein the 
comparison means includes a new paragraph detector to determine 
whether an input word is the first word of a new paragraph. 

10. A system as claimed in any preceding claim, wherein the means 
(41) for formatting includes means for aligning the handwritten image 
on parallel horizontal lines. 
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11. A system as claimed in claim 10 , wherein the means for aligning 
includes first means (58) for orientating an input word to be 
horizontal with respect to a predetermined horizontal orientation, 
and second means (54) for aligning horizontal words on a common 
horizontal line. 

12. A system as claimed in claim 11, wherein the first means 
includes a word length detector (50) for determining whether an input 
word is longer than a predetermined length and a word baseline 
determinator for establishing whether or not a baseline for a 
sequence of words exceeds the predetermined length. 

13. A system as claimed in claim 12, wherein the second means 
includes a first register (53) for storing baselines of input words 
from the baselines determinator, and a second register (56) storing a 
running baseline. 

14. A system as claimed in claim 10, wherein the means for aligning 
the handwritten image includes 

a word length detector (50) receiving an input handwritten 
image, 

means (52) to determine whether the word of the handwritten 
image contains any descender characters, 

a register (56) to store a running baseline representative of 
the horizontal alignment of words of the handwritten image, 

means (53) to establish the word baseline of the input 
handwritten image, 

means (54) responsive to the running baseline, descender 
information and the word baseline to correct the running 
baseline as a function of the word baseline, and 
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means (58) to transform the position of the input handwritten 
image as a function of the corrected running baseline. 

15. A system as claimed in any preceding claim, further including 
means to edit the handwritten image. 

16. A system as claimed in claim 15, wherein the X-Y tablet includes 
an interactive zone (82) responsive to the input stylus for 
generating edit commands to the means for formatting and a zone for 
entering replacement words. 

17. A system as claimed in claim 15, wherein the means to edit 
includes t 

edit co-ordinate analyser (61) means for determining the 
functional command to be performed, 

means (65) for storing an operand of the functional command to 
be performed, 

logic means (62, 63 and 64) for performing the operand, and 
means (47) for storing the edited image. 

18. A system as claimed in any of claims 15 to 17, wherein the means 
(67 and 70) to edit includes means to shift segmented words of the 
handwritten image into locations on the display means that are 
positionally different from the positions of direct mapping of the 
x-y tablet onto the display. 

19. A system as claimed in any of claims 15 to 18, wherein the means 
(66 and 69) to edit includes means to delete portions of the 
handwritten image. 
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